{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML with Tabular data - Using AutoGluon \n",
    "\n",
    "We first demonstrate how to easily use AutoGluon for classification/regression problems with data stored in tables. Here, we use a dataset of diabetes patients at a hospital with various medical features recorded for each patient (described in detail [here](https://archive.ics.uci.edu/ml/datasets/diabetes+130-us+hospitals+for+years+1999-2008)). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Via a simple fit() call, AutoGluon can produce highly-accurate models to predict the values in one column of a data table based on the rest of the columns’ values. \n",
    "To start, we import AutoGluon and specify that *TabularPrediction* is our task of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogluon import TabularPrediction as task\n",
    "\n",
    "# Other packages used in this notebook:\n",
    "import pprint\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "\n",
    "subsample_size = 600 # experiment with larger values to try AutoGluon with larger datasets "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load training data from a CSV file into an AutoGluon Dataset object. This object is essentially equivalent to a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) and the same methods can be applied to both. To ensure fast runtimes, we subsample all datasets here, but encourage you to increase `subsample_size` above to see how AutoGluon behaves on more realistically-sized datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv | Columns = 47 / 47 | Rows = 61059 -> 61059\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>payer_code</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>...</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "      <th>readmitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Female</td>\n",
       "      <td>[0-10)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"25\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>1.5</td>\n",
       "      <td>?</td>\n",
       "      <td>Pediatrics-Endocrinology</td>\n",
       "      <td>41</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Female</td>\n",
       "      <td>[10-20)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>3.0</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>59</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Female</td>\n",
       "      <td>[20-30)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>2.3</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Male</td>\n",
       "      <td>[30-40)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>2.3</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>44</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Male</td>\n",
       "      <td>[40-50)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>1.7</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   gender      age weight admission_type_id discharge_disposition_id  \\\n",
       "0  Female   [0-10)      ?               \"6\"                     \"25\"   \n",
       "1  Female  [10-20)      ?               \"1\"                      \"1\"   \n",
       "2  Female  [20-30)      ?               \"1\"                      \"1\"   \n",
       "3    Male  [30-40)      ?               \"1\"                      \"1\"   \n",
       "4    Male  [40-50)      ?               \"1\"                      \"1\"   \n",
       "\n",
       "  admission_source_id  time_in_hospital payer_code         medical_specialty  \\\n",
       "0                 \"1\"               1.5          ?  Pediatrics-Endocrinology   \n",
       "1                 \"7\"               3.0          ?                         ?   \n",
       "2                 \"7\"               2.3          ?                         ?   \n",
       "3                 \"7\"               2.3          ?                         ?   \n",
       "4                 \"7\"               1.7          ?                         ?   \n",
       "\n",
       "   num_lab_procedures  ...  citoglipton  insulin  glyburide-metformin  \\\n",
       "0                  41  ...           No       No                   No   \n",
       "1                  59  ...           No       Up                   No   \n",
       "2                  11  ...           No       No                   No   \n",
       "3                  44  ...           No       Up                   No   \n",
       "4                  51  ...           No   Steady                   No   \n",
       "\n",
       "   glipizide-metformin  glimepiride-pioglitazone metformin-rosiglitazone  \\\n",
       "0                   No                        No                      No   \n",
       "1                   No                        No                      No   \n",
       "2                   No                        No                      No   \n",
       "3                   No                        No                      No   \n",
       "4                   No                        No                      No   \n",
       "\n",
       "  metformin-pioglitazone change  diabetesMed readmitted  \n",
       "0                     No     No           No         NO  \n",
       "1                     No     Ch          Yes        >30  \n",
       "2                     No     No          Yes         NO  \n",
       "3                     No     Ch          Yes         NO  \n",
       "4                     No     Ch          Yes         NO  \n",
       "\n",
       "[5 rows x 47 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv')\n",
    "train_data = train_data.head(subsample_size) # subsample data for faster demo\n",
    "display(train_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we loaded data from a CSV file stored in the cloud (AWS s3 bucket), but you can you specify a local file-path instead if you have already downloaded the CSV file to your own machine (e.g., using `wget`). Each row in the table train_data corresponds to a single training example. In this particular dataset, each row corresponds to an individual person, and the columns contain various facts about the patient and their current hospital visit.\n",
    "\n",
    "We use these features to predict whether each patient will be readmitted to the hospital within <30 days, after >30 days, or not at all (NO), which is recorded in the **readmitted** column of this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Possible values of target variable (with occurrence counts): \n",
      " NO     302\n",
      ">30    247\n",
      "<30     51\n"
     ]
    }
   ],
   "source": [
    "label_column = 'readmitted'\n",
    "print(\"Possible values of target variable (with occurrence counts): \\n\", train_data[label_column].value_counts().to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how easy it is to use AutoGluon to tackle this multiclass classification problem (with 3 possible classes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ...\n",
      "AutoGluon will save models to agModels-predictReadmit/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 600 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.44s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini ...\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t0.96s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr ...\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t0.81s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini ...\n",
      "\t0.5667\t = Validation accuracy score\n",
      "\t0.74s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr ...\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t0.72s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif ...\n",
      "\t0.5\t = Validation accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist ...\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier ...\n",
      "\t0.65\t = Validation accuracy score\n",
      "\t6.1s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier ...\n",
      "\t0.6167\t = Validation accuracy score\n",
      "\t42.82s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier ...\n",
      "\t0.5083\t = Validation accuracy score\n",
      "\t6.78s\t = Training runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifierCustom ...\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t2.43s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ...\n",
      "\t0.6583\t = Validation accuracy score\n",
      "\t0.48s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 64.87s ...\n"
     ]
    }
   ],
   "source": [
    "dir = 'agModels-predictReadmit' # specifies folder where to store trained models\n",
    "predictor = task.fit(train_data=train_data, label=label_column, output_directory=dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, load separate test data to demonstrate how to make predictions for new patients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv | Columns = 47 / 47 | Rows = 20354 -> 20354\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>payer_code</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>...</th>\n",
       "      <th>examide</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Male</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>2.6</td>\n",
       "      <td>BC</td>\n",
       "      <td>?</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Female</td>\n",
       "      <td>[90-100)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>6.1</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>64</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Female</td>\n",
       "      <td>[50-60)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"3\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>4.5</td>\n",
       "      <td>HM</td>\n",
       "      <td>?</td>\n",
       "      <td>29</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Female</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>4.0</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>43</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Male</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"3\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>1.3</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   gender       age weight admission_type_id discharge_disposition_id  \\\n",
       "0    Male   [60-70)      ?               \"2\"                      \"1\"   \n",
       "1  Female  [90-100)      ?               \"2\"                      \"6\"   \n",
       "2  Female   [50-60)      ?               \"3\"                      \"1\"   \n",
       "3  Female   [70-80)      ?               \"1\"                      \"1\"   \n",
       "4    Male   [60-70)      ?               \"3\"                      \"1\"   \n",
       "\n",
       "  admission_source_id  time_in_hospital payer_code medical_specialty  \\\n",
       "0                 \"1\"               2.6         BC                 ?   \n",
       "1                 \"1\"               6.1         MC                 ?   \n",
       "2                 \"1\"               4.5         HM                 ?   \n",
       "3                 \"7\"               4.0         MC                 ?   \n",
       "4                 \"1\"               1.3         MC                 ?   \n",
       "\n",
       "   num_lab_procedures  ...  examide  citoglipton  insulin  \\\n",
       "0                  10  ...       No           No       No   \n",
       "1                  64  ...       No           No       No   \n",
       "2                  29  ...       No           No       No   \n",
       "3                  43  ...       No           No   Steady   \n",
       "4                  12  ...       No           No       No   \n",
       "\n",
       "   glyburide-metformin  glipizide-metformin glimepiride-pioglitazone  \\\n",
       "0                   No                   No                       No   \n",
       "1                   No                   No                       No   \n",
       "2                   No                   No                       No   \n",
       "3                   No                   No                       No   \n",
       "4                   No                   No                       No   \n",
       "\n",
       "  metformin-rosiglitazone metformin-pioglitazone  change diabetesMed  \n",
       "0                      No                     No      No          No  \n",
       "1                      No                     No      Ch         Yes  \n",
       "2                      No                     No      No          No  \n",
       "3                      No                     No      Ch         Yes  \n",
       "4                      No                     No      No         Yes  \n",
       "\n",
       "[5 rows x 46 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv')\n",
    "test_data = test_data.head(subsample_size) # subsample data for faster demo\n",
    "y_test = test_data[label_column] # ground-truth target values\n",
    "test_data_nolab = test_data.drop(labels=[label_column],axis=1) # delete label column to prove we're not cheating\n",
    "display(test_data_nolab.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use AutoGluon's trained models to make predictions on the new data and evaluate their performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: accuracy on test data: 0.5233333333333333\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.5233333333333333,\n",
      "    \"accuracy_score\": 0.5233333333333333,\n",
      "    \"balanced_accuracy_score\": 0.371436481569372,\n",
      "    \"matthews_corrcoef\": 0.09290760510430489\n",
      "}\n",
      "Detailed (per-class) classification report:\n",
      "{\n",
      "    \"<30\": {\n",
      "        \"precision\": 0.13043478260869565,\n",
      "        \"recall\": 0.05,\n",
      "        \"f1-score\": 0.07228915662650602,\n",
      "        \"support\": 60\n",
      "    },\n",
      "    \">30\": {\n",
      "        \"precision\": 0.37435897435897436,\n",
      "        \"recall\": 0.37244897959183676,\n",
      "        \"f1-score\": 0.37340153452685426,\n",
      "        \"support\": 196\n",
      "    },\n",
      "    \"NO\": {\n",
      "        \"precision\": 0.6230366492146597,\n",
      "        \"recall\": 0.6918604651162791,\n",
      "        \"f1-score\": 0.6556473829201103,\n",
      "        \"support\": 344\n",
      "    },\n",
      "    \"accuracy\": 0.5233333333333333,\n",
      "    \"macro avg\": {\n",
      "        \"precision\": 0.3759434687274433,\n",
      "        \"recall\": 0.371436481569372,\n",
      "        \"f1-score\": 0.36711269135782354,\n",
      "        \"support\": 600\n",
      "    },\n",
      "    \"weighted avg\": {\n",
      "        \"precision\": 0.49254175543453943,\n",
      "        \"recall\": 0.5233333333333333,\n",
      "        \"f1-score\": 0.5051112498156196,\n",
      "        \"support\": 600\n",
      "    }\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions:\n",
      " ['NO' 'NO' '<30' '<30' 'NO' '>30' '>30' 'NO' '>30' 'NO' 'NO' 'NO' '>30'\n",
      " 'NO' 'NO' 'NO' '<30' 'NO' '>30' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO'\n",
      " '>30' '>30' 'NO' '>30' '>30' '>30' 'NO' 'NO' 'NO' 'NO' '>30' '>30' '>30'\n",
      " 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '<30' 'NO' 'NO' '>30' 'NO' '>30' '>30'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' '>30' '>30' 'NO' '>30' '<30' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' 'NO' '>30' 'NO' '>30' 'NO' '>30' 'NO' 'NO' '>30' '>30' 'NO'\n",
      " '>30' 'NO' 'NO' 'NO' '>30' '>30' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO'\n",
      " '<30' 'NO' 'NO' '>30' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '>30' 'NO' '>30' 'NO' '>30' '>30'\n",
      " 'NO' '>30' 'NO' 'NO' '>30' '>30' '>30' '>30' 'NO' 'NO' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' 'NO' '>30'\n",
      " 'NO' '>30' 'NO' '>30' 'NO' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' '>30'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' '<30' '>30'\n",
      " '>30' '>30' '>30' 'NO' '>30' '>30' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO'\n",
      " '>30' '>30' 'NO' 'NO' '>30' '>30' '<30' 'NO' '>30' '>30' 'NO' '<30' '>30'\n",
      " 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '>30' 'NO' '>30'\n",
      " 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' '>30' 'NO' '>30' 'NO' '>30' 'NO'\n",
      " 'NO' '>30' 'NO' 'NO' '>30' 'NO' 'NO' '<30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' 'NO' '>30' 'NO' '>30' 'NO' 'NO' 'NO' '>30' 'NO'\n",
      " '>30' 'NO' '>30' 'NO' 'NO' 'NO' '>30' '<30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '>30' 'NO'\n",
      " '>30' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' 'NO'\n",
      " 'NO' 'NO' '>30' '>30' 'NO' 'NO' '>30' '>30' 'NO' 'NO' '>30' '>30' '>30'\n",
      " '>30' 'NO' 'NO' '>30' 'NO' 'NO' '>30' 'NO' '>30' '>30' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '<30'\n",
      " 'NO' '>30' 'NO' 'NO' '>30' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO'\n",
      " 'NO' '>30' 'NO' '>30' 'NO' 'NO' '>30' '>30' 'NO' 'NO' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO'\n",
      " '<30' 'NO' 'NO' 'NO' 'NO' 'NO' '<30' '>30' 'NO' 'NO' '>30' 'NO' '>30'\n",
      " '>30' 'NO' '<30' 'NO' '>30' '>30' '>30' '>30' 'NO' '>30' 'NO' 'NO' 'NO'\n",
      " '<30' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' '<30' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '>30'\n",
      " 'NO' '>30' 'NO' '<30' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' '>30' 'NO' 'NO' '>30' 'NO' 'NO' '>30' '>30' 'NO' 'NO' '>30'\n",
      " '>30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30' 'NO' '>30'\n",
      " 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' '>30' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30' '>30' 'NO'\n",
      " 'NO' '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' '>30' '<30'\n",
      " '>30' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' '>30' '>30' '>30' 'NO' 'NO' 'NO' '>30' 'NO' 'NO' 'NO' '>30'\n",
      " '<30' 'NO' 'NO' 'NO' 'NO' '<30' 'NO' 'NO' 'NO' 'NO' 'NO' 'NO' '<30' 'NO'\n",
      " 'NO' 'NO' '>30' '<30' 'NO' '>30' '>30' 'NO' '>30' 'NO' 'NO']\n"
     ]
    }
   ],
   "source": [
    "predictor = task.load(dir) # unnecessary, just demonstrates how to load previously-trained predictor from file\n",
    "\n",
    "y_pred = predictor.predict(test_data_nolab)\n",
    "perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)\n",
    "print(\"Predictions:\\n\", y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression (predicting numeric table columns):\n",
    "\n",
    "To briefly demonstrate that AutoGluon's `fit()` can also automatically handle regression tasks, we now try to predict the numeric **time_in_hospital** variable in the same table based on the other features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVhUlEQVR4nO3df5TddX3n8edriT+AcQlIndIEO6xltSxpVWb9xZ7uRNouChXOHmqxLA0WT45dq2zNHo3tbt3tqS2eLlp7bOvJisK2lGgRDxSklUWmnu4KbYLWCGiJGH5ETFQgGmSrWd/7x3xxh8kkM3Pv3LkzH56Pc3Lm3u+Pz/c1NzOv+c7n3vudVBWSpLb8k2EHkCQtPstdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlruWVJLnJtmf5IglPOYHkvznPse4IslvL1ameR7zpiQbetx3LEklWbXYubQy+B+vgUuyC3hDVf3PqrofGFnK41fVG5fyeIulql71xO0kFzH1GP6r4SXSSuKZuyQ1yHLXQCX5E+C5wF900zFvmz5dkGQyyW8n+d/d+r9I8uwkVyX5VpK/SzI2bbwXJLk5ycNJvpTktfPI8IMplSQTSR5MsinJ3iQPJXn9PD+dY5PcmOTbSW5P8rxpx3hFl3Vf9/EV09ZdlOTebr+vJLlg2vL/leT93X5fTHLGtP0mk7whyY8DHwBe3j1Gj3brz0ry2e5xeiDJf5nn56GnAMtdA1VVFwL3Az9XVSPAR2fZ7HzgQmAN8DzgM8CHgeOAu4F3AiQ5GrgZ+DPgOd1+f5TklAXG+mHgmO54FwN/mOTYeex3PvBfgWOBncC7ulzHATcCfwA8G3gPcGP3Q+robvmrqupZwCuAz00b86XAl4Hju8/z2m68H6iqu4E3Ap+pqpGqWt2tegz4JWA1cBbwK0nOne+DoLZZ7loOPlxVX66qfcBNwJe7+fkDwJ8DL+q2OxvYVVUfrqoDVfVZ4GPAzy/weN8DfquqvldVnwD2A8+fx34fr6q/7XJdBbywW34WcE9V/UmX62rgi8DPdeu/D5ya5Miqeqiq7pw25l7g97ssHwG+1I03p6qarKodVfX9qvo8cDXwr+ezr9pnuWs52DPt9uOz3H/iCdgfBV6a5NEn/gEXMHUmvhDf7Ar6Cd9hfk/yfu0Q+/wIcN+Mbe8D1lTVY8AvMHXm/VA3rfOCadvtridfve++brw5JXlpkluTfD3Jvu4Yx89nX7XPctdSWKxLjz4A/HVVrZ72b6SqfmWRxu/VV5n6wTPdc4HdAFX1V1X1M8AJTJ3R//dp261Jkhn7fXWWY8z2GP4ZcD1wYlUdw9S8fGbZTk9BlruWwh7gny3CODcA/zzJhUme1v37l90TjsP0iS7XLyZZleQXgFOAG5KMJjmnm3v/R6amgL4/bd/nAG/pPpefB368G2+mPcDaJE+ftuxZwMNV9X+SvAT4xQF8blqhLHcthd8F/lM3jXJer4NU1beBn2Xqic2vMjVN8m7gGYuQsWdV9U2mng/YBHwTeBtwdlV9g6nvsbcylfdhpubEp/+mcTtwMvANpp6gPa8bb6ZPAXcCX0vyjW7Zvwd+K8m3gd9k9ier9RQV/1iHNBy+MUmD5Jm7JDXIclcTktzZvcFn5r8LlnIMablwWkaSGuSZuyQ1aFlcFfL444+vsbGxYcd4kscee4yjjz562DHmzbyDtZLyrqSsYN5+bN++/RtV9UOzrVsW5T42Nsa2bduGHeNJJicnmZiYGHaMeTPvYK2kvCspK5i3H0lmvjP6B5yWkaQGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBi2Ld6j2Y2zzjQMZd9O6A1w0x9i7Lp3X3zGWpCW34st9mAb1g2Uu/lCRNBenZSSpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkho0Z7kn+VCSvUm+MG3Z7yX5YpLPJ/l4ktXT1r0jyc4kX0rybwaUW5J0GPM5c78COHPGspuBU6vqJ4B/AN4BkOQU4HzgX3T7/FGSIxYtrSRpXuYs96r6NPDwjGWfrKoD3d3bgLXd7XOArVX1j1X1FWAn8JJFzCtJmofFmHP/ZeCm7vYa4IFp6x7slkmSllCqau6NkjHghqo6dcby3wDGgX9bVZXk/cBtVfWn3frLgZuq6ppZxtwIbAQYHR09bevWrT19Ajt27+tpv7mMHgl7Hh/I0H1bt+aYg5bt37+fkZGRIaTpjXkHZyVlBfP2Y/369durany2dT3/mb0kFwFnA2fU//8JsRs4cdpma7tlB6mqLcAWgPHx8ZqYmOgpx1x/57RXm9Yd4LIdy/OvEO66YOKgZZOTk/T6GA6DeQdnJWUF8w5KT9MySc4E3ga8pqq+M23V9cD5SZ6R5CTgZOBv+48pSVqIOU9Nk1wNTADHJ3kQeCdTr455BnBzEpiainljVd2Z5KPAXcAB4E1V9X8HFV6SNLs5y72qXjfL4ssPs/27gHf1E0qS1B/foSpJDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSg+Ys9yQfSrI3yRemLTsuyc1J7uk+HtstT5I/SLIzyeeTvHiQ4SVJs5vPmfsVwJkzlm0Gbqmqk4FbuvsArwJO7v5tBP54cWJKkhZiznKvqk8DD89YfA5wZXf7SuDcacv/R025DVid5IRFyipJmqdU1dwbJWPADVV1anf/0apa3d0O8EhVrU5yA3BpVf1Nt+4W4O1VtW2WMTcydXbP6OjoaVu3bu3pE9ixe19P+81l9EjY8/hAhu7bujXHHLRs//79jIyMDCFNb8w7OCspK5i3H+vXr99eVeOzrVvV7+BVVUnm/glx8H5bgC0A4+PjNTEx0dPxL9p8Y0/7zWXTugNctqPvh2cgdl0wcdCyyclJen0Mh8G8g7OSsoJ5B6XXV8vseWK6pfu4t1u+Gzhx2nZru2WSpCXUa7lfD2zobm8Arpu2/Je6V828DNhXVQ/1mVGStEBzzjskuRqYAI5P8iDwTuBS4KNJLgbuA17bbf4J4NXATuA7wOsHkFmSNIc5y72qXneIVWfMsm0Bb+o3lCSpP75DVZIaZLlLUoMsd0lq0PJ8IbcOa2yW1/ZvWndgYK/5n27XpWcN/BiS+ueZuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBvVV7kl+LcmdSb6Q5Ookz0xyUpLbk+xM8pEkT1+ssJKk+em53JOsAd4CjFfVqcARwPnAu4H3VtWPAY8AFy9GUEnS/PU7LbMKODLJKuAo4CHglcA13forgXP7PIYkaYFSVb3vnFwCvAt4HPgkcAlwW3fWTpITgZu6M/uZ+24ENgKMjo6etnXr1p4y7Ni9r7fwcxg9EvY8PpChB2Kp8q5bc8yijLN//35GRkYWZaylsJLyrqSsYN5+rF+/fntVjc+2blWvgyY5FjgHOAl4FPhz4Mz57l9VW4AtAOPj4zUxMdFTjos239jTfnPZtO4Al+3o+eFZckuVd9cFE4syzuTkJL3+nw/DSsq7krKCeQeln2mZnwa+UlVfr6rvAdcCpwOru2kagLXA7j4zSpIWqJ9yvx94WZKjkgQ4A7gLuBU4r9tmA3BdfxElSQvVc7lX1e1MPXF6B7CjG2sL8HbgrUl2As8GLl+EnJKkBehrkraq3gm8c8bie4GX9DOuJKk/vkNVkhpkuUtSgyx3SWqQ5S5JDVo579LRsjC2SG8a27TuwILegLbr0rMW5bjSU4Vn7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGtRXuSdZneSaJF9McneSlyc5LsnNSe7pPh67WGElSfPT75n7+4C/rKoXAD8J3A1sBm6pqpOBW7r7kqQl1HO5JzkG+CngcoCq+m5VPQqcA1zZbXYlcG5/ESVJC5Wq6m3H5IXAFuAups7atwOXALuranW3TYBHnrg/Y/+NwEaA0dHR07Zu3dpTjh279/W031xGj4Q9jw9k6IFoPe+6NccMLsw87N+/n5GRkaFmmK+VlBXM24/169dvr6rx2db1U+7jwG3A6VV1e5L3Ad8C3jy9zJM8UlWHnXcfHx+vbdu29ZRjbPONPe03l03rDnDZjlUDGXsQWs+769KzBphmbpOTk0xMTAw1w3ytpKxg3n4kOWS59zPn/iDwYFXd3t2/BngxsCfJCd2BTwD29nEMSVIPei73qvoa8ECS53eLzmBqiuZ6YEO3bANwXV8JJUkL1u/v8W8GrkrydOBe4PVM/cD4aJKLgfuA1/Z5DEnSAvVV7lX1OWC2+Z4z+hlXktSflfMMnJ7SBvXE+XwM+8lcqRdefkCSGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQX2Xe5Ijknw2yQ3d/ZOS3J5kZ5KPJHl6/zElSQuxGGfulwB3T7v/buC9VfVjwCPAxYtwDEnSAvRV7knWAmcBH+zuB3glcE23yZXAuf0cQ5K0cKmq3ndOrgF+F3gW8B+Bi4DburN2kpwI3FRVp86y70ZgI8Do6OhpW7du7SnDjt37etpvLqNHwp7HBzL0QJh3cNatOYb9+/czMjIy7CjzspKygnn7sX79+u1VNT7bulW9DprkbGBvVW1PMrHQ/atqC7AFYHx8vCYmFjwEABdtvrGn/eayad0BLtvR88Oz5Mw7OLsumGBycpJev0aX2krKCuYdlH6+u04HXpPk1cAzgX8KvA9YnWRVVR0A1gK7+48pDc/Y5hvZtO7AwE4kDmXXpWct6fHUlp7n3KvqHVW1tqrGgPOBT1XVBcCtwHndZhuA6/pOKUlakEG8zv3twFuT7ASeDVw+gGNIkg5jUSY9q2oSmOxu3wu8ZDHGlST1xneoSlKDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWrQyrjmqvQUNNbjVSj7vYKlV6Nsg2fuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSg3ou9yQnJrk1yV1J7kxySbf8uCQ3J7mn+3js4sWVJM1HP2fuB4BNVXUK8DLgTUlOATYDt1TVycAt3X1J0hLqudyr6qGquqO7/W3gbmANcA5wZbfZlcC5fWaUJC1Qqqr/QZIx4NPAqcD9VbW6Wx7gkSfuz9hnI7ARYHR09LStW7f2dOwdu/f1tN9cRo+EPY8PZOiBMO9graS8/WZdt+aYxQszD/v372dkZGRJj9mP5ZR3/fr126tqfLZ1fZd7khHgr4F3VdW1SR6dXuZJHqmqw867j4+P17Zt23o6fq/XvJ7LpnUHuGzHyrncvXkHayXl7TfrUl/PfXJykomJiSU9Zj+WU94khyz3vl4tk+RpwMeAq6rq2m7xniQndOtPAPb2cwxJ0sL182qZAJcDd1fVe6atuh7Y0N3eAFzXezxJUi/6+T3zdOBCYEeSz3XLfh24FPhokouB+4DX9pVQ0pIa1FTnofT7ZwEXQ4t/WrDncq+qvwFyiNVn9DquJKl/vkNVkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQSvjT8tI0gAt5DLHi32J4kFdbtgzd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJatDAyj3JmUm+lGRnks2DOo4k6WADKfckRwB/CLwKOAV4XZJTBnEsSdLBBnXm/hJgZ1XdW1XfBbYC5wzoWJKkGVJViz9och5wZlW9obt/IfDSqvrVadtsBDZ2d58PfGnRg/TneOAbww6xAOYdrJWUdyVlBfP240er6odmWzG0q0JW1RZgy7COP5ck26pqfNg55su8g7WS8q6krGDeQRnUtMxu4MRp99d2yyRJS2BQ5f53wMlJTkrydOB84PoBHUuSNMNApmWq6kCSXwX+CjgC+FBV3TmIYw3Qsp0yOgTzDtZKyruSsoJ5B2IgT6hKkobLd6hKUoMsd0lqkOU+Q5ITk9ya5K4kdya5ZNiZ5pLkiCSfTXLDsLPMJcnqJNck+WKSu5O8fNiZDifJr3VfB19IcnWSZw4703RJPpRkb5IvTFt2XJKbk9zTfTx2mBmnO0Te3+u+Hj6f5ONJVg8x4pPMlnfauk1JKsnxw8g2F8v9YAeATVV1CvAy4E0r4NIJlwB3DzvEPL0P+MuqegHwkyzj3EnWAG8BxqvqVKZeHHD+cFMd5ArgzBnLNgO3VNXJwC3d/eXiCg7OezNwalX9BPAPwDuWOtRhXMHBeUlyIvCzwP1LHWi+LPcZquqhqrqju/1tpspnzXBTHVqStcBZwAeHnWUuSY4Bfgq4HKCqvltVjw411NxWAUcmWQUcBXx1yHmepKo+DTw8Y/E5wJXd7SuBc5cy0+HMlreqPllVB7q7tzH1vphl4RCPL8B7gbcBy/YVKZb7YSQZA14E3D7kKIfz+0x9kX1/yDnm4yTg68CHu2mkDyY5etihDqWqdgP/jamzs4eAfVX1yeGmmpfRqnqou/01YHSYYRbol4Gbhh3icJKcA+yuqr8fdpbDsdwPIckI8DHgP1TVt4adZzZJzgb2VtX2YWeZp1XAi4E/rqoXAY+xvKYMnqSbqz6HqR9KPwIcneTfDTfVwtTUa52X7dnldEl+g6lp0auGneVQkhwF/Drwm8POMhfLfRZJnsZUsV9VVdcOO89hnA68Jskupq68+cokfzrcSIf1IPBgVT3xm9A1TJX9cvXTwFeq6utV9T3gWuAVQ840H3uSnADQfdw75DxzSnIRcDZwQS3vN988j6kf9n/ffd+tBe5I8sNDTTULy32GJGFqTvjuqnrPsPMcTlW9o6rWVtUYU0/0faqqlu2ZZVV9DXggyfO7RWcAdw0x0lzuB16W5Kju6+IMlvETwNNcD2zobm8ArhtiljklOZOpqcXXVNV3hp3ncKpqR1U9p6rGuu+7B4EXd1/by4rlfrDTgQuZOgv+XPfv1cMO1ZA3A1cl+TzwQuB3hhvn0LrfMK4B7gB2MPX9sqzeep7kauAzwPOTPJjkYuBS4GeS3MPUbx+XDjPjdIfI+37gWcDN3ffbB4YacppD5F0RvPyAJDXIM3dJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhr0/wBzToyVoG5+DQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "time_column = 'time_in_hospital'\n",
    "hist_plot = train_data.hist(column=time_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We again call `fit()`, imposing a time-limit this time (in seconds), and also demonstrate a shorthand method to evaluate the resulting model's predictive performance on the test data (which here still contain the label column):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to agModels-predictTime/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the first 10 unique label values in your data:  [1.5, 3.0, 2.3, 1.7000000000000002, 3.9, 4.3, 5.5, 13.2, 12.4, 9.7]\n",
      "AutoGluon infers your prediction problem is: regression  (because dtype of label-column == float and many unique label-values observed).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Feature Generator processed 600 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 26\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 26\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tint features: 7\n",
      "\tcategory features: 26\n",
      "Processed Features:\n",
      "\tint features: 7\n",
      "\tcategory features: 26\n",
      "\tData preprocessing and feature engineering runtime = 0.21s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: root_mean_squared_error\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: root_mean_squared_error\n",
      "Fitting model: RandomForestRegressorMSE ... Training model for up to 29.79s of the 29.79s of remaining time.\n",
      "\t-2.5482\t = Validation root_mean_squared_error score\n",
      "\t0.81s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesRegressorMSE ... Training model for up to 28.8s of the 28.8s of remaining time.\n",
      "\t-2.4621\t = Validation root_mean_squared_error score\n",
      "\t0.58s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: KNeighborsRegressorUnif ... Training model for up to 27.97s of the 27.97s of remaining time.\n",
      "\t-2.7959\t = Validation root_mean_squared_error score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsRegressorDist ... Training model for up to 27.85s of the 27.85s of remaining time.\n",
      "\t-2.754\t = Validation root_mean_squared_error score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.1s\t = Validation runtime\n",
      "Fitting model: LightGBMRegressor ... Training model for up to 27.72s of the 27.72s of remaining time.\n",
      "\t-2.3473\t = Validation root_mean_squared_error score\n",
      "\t0.37s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: CatboostRegressor ... Training model for up to 27.31s of the 27.31s of remaining time.\n",
      "\t-2.4206\t = Validation root_mean_squared_error score\n",
      "\t4.97s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: NeuralNetRegressor ... Training model for up to 22.29s of the 22.29s of remaining time.\n",
      "\t-2.45\t = Validation root_mean_squared_error score\n",
      "\t12.1s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: LightGBMRegressorCustom ... Training model for up to 10.11s of the 10.11s of remaining time.\n",
      "\t-2.5225\t = Validation root_mean_squared_error score\n",
      "\t2.27s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 29.79s of the 7.0s of remaining time.\n",
      "\t-2.3052\t = Validation root_mean_squared_error score\n",
      "\t0.48s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 23.55s ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictive performance on given dataset: root_mean_squared_error = 2.5721178269000005\n"
     ]
    }
   ],
   "source": [
    "predictor_time = task.fit(train_data=train_data, output_directory=\"agModels-predictTime\", \n",
    "                          label=time_column, time_limits=30)\n",
    "performance = predictor_time.evaluate(test_data) # another way of gauging predictor accuracy besides `evaluate_predictions()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we didn’t even need to tell AutoGluon this is a regression problem, it automatically inferred this from the data and reported an appropriate performance metric (RMSE by default). For evaluation metrics where higher values are worse (like RMSE), AutoGluon may sometimes flips their sign and print them as *negative* values during training (as it internally assumes higher values are better)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of fit():\n",
    "    \n",
    "Let's revisit our original `predictor` (for predicting the **readmitted** variable) and see what happened during `fit()`.\n",
    "\n",
    "Since there are only three possible values of the class variable, this was a 3-class classification problem, for which an appropriate performance metric is accuracy. \n",
    "AutoGluon automatically infers this if unspecified, but you can explicitly specify these things via the `problem_type` and  `eval_metric` [arguments of `fit()`](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit) and AutoGluon will tailor its models to optimize your metric. \n",
    "AutoGluon also infers the type of each feature (i.e. which columns contain continuous numbers vs. discrete categories) and can automatically handle common issues like missing data and normalizing numerical features as necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluon infers prediction problem is of type: multiclass\n",
      "AutoGluon assigned the features to the following types:\n",
      "{'category': ['gender',\n",
      "              'age',\n",
      "              'admission_type_id',\n",
      "              'discharge_disposition_id',\n",
      "              'admission_source_id',\n",
      "              'medical_specialty',\n",
      "              'diag_1',\n",
      "              'diag_2',\n",
      "              'diag_3',\n",
      "              'max_glu_serum',\n",
      "              'A1Cresult',\n",
      "              'metformin',\n",
      "              'repaglinide',\n",
      "              'glimepiride',\n",
      "              'glipizide',\n",
      "              'glyburide',\n",
      "              'tolbutamide',\n",
      "              'pioglitazone',\n",
      "              'rosiglitazone',\n",
      "              'acarbose',\n",
      "              'troglitazone',\n",
      "              'tolazamide',\n",
      "              'insulin',\n",
      "              'change',\n",
      "              'diabetesMed'],\n",
      " 'float': ['time_in_hospital'],\n",
      " 'int': ['num_lab_procedures',\n",
      "         'num_procedures',\n",
      "         'num_medications',\n",
      "         'number_outpatient',\n",
      "         'number_emergency',\n",
      "         'number_inpatient',\n",
      "         'number_diagnoses']}\n"
     ]
    }
   ],
   "source": [
    "print(\"AutoGluon infers prediction problem is of type:\", predictor.problem_type)\n",
    "print(\"AutoGluon assigned the features to the following types:\")\n",
    "pprint.pprint(dict(predictor.feature_types.feature_types_raw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We did not specify separate validation data and so AutoGluon automatically chooses a random training/validation split of the data. The data used for validation is seperated from the training data and is used to determine the models (and possibly hyperparameter-values) that produce the best results.  Rather than just a single model, AutoGluon trains multiple models and ensembles them together to ensure superior predictive performance.  By default, AutoGluon tries to fit various types of models including neural networks and various sorts of tree ensembles. \n",
    "\n",
    "For TabularPrediction tasks, `fit()` returns a Predictor object. Besides inference, this object can also be used to view a summary of what happened during `fit()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Summary of fit() ***\n",
      "Estimated performance of each model:\n",
      "                         model  score_val  pred_time_val   fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer\n",
      "0      weighted_ensemble_k0_l1   0.658333       0.165535   7.317772                0.001362           0.479501            1       True\n",
      "1           LightGBMClassifier   0.650000       0.031892   6.098786                0.031892           6.098786            0       True\n",
      "2           CatboostClassifier   0.616667       0.036838  42.824128                0.036838          42.824128            0       True\n",
      "3   RandomForestClassifierEntr   0.608333       0.139243   0.808625                0.139243           0.808625            0       True\n",
      "4     LightGBMClassifierCustom   0.600000       0.033570   2.430807                0.033570           2.430807            0       True\n",
      "5   RandomForestClassifierGini   0.591667       0.131530   0.956434                0.131530           0.956434            0       True\n",
      "6     ExtraTreesClassifierEntr   0.591667       0.141177   0.723158                0.141177           0.723158            0       True\n",
      "7     ExtraTreesClassifierGini   0.566667       0.132281   0.739485                0.132281           0.739485            0       True\n",
      "8     KNeighborsClassifierDist   0.533333       0.105541   0.004107                0.105541           0.004107            0       True\n",
      "9          NeuralNetClassifier   0.508333       0.076282   6.777301                0.076282           6.777301            0       True\n",
      "10    KNeighborsClassifierUnif   0.500000       0.107808   0.006475                0.107808           0.006475            0       True\n",
      "Number of models trained: 11\n",
      "Types of models trained:\n",
      "{'CatboostModel', 'KNNModel', 'RFModel', 'XTModel', 'WeightedEnsembleModel', 'LGBModel', 'TabularNeuralNetModel'}\n",
      "Bagging used: False \n",
      "Stack-ensembling used: False \n",
      "Hyperparameter-tuning used: False \n",
      "User-specified hyperparameters:\n",
      "{'default': {'NN': [{}], 'GBM': [{}], 'CAT': [{}], 'RF': [{'criterion': 'gini', 'AG_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'AG_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}], 'XT': [{'criterion': 'gini', 'AG_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'AG_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}], 'KNN': [{'weights': 'uniform', 'AG_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'AG_args': {'name_suffix': 'Dist'}}], 'custom': [{'num_boost_round': 10000, 'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'learning_rate': 0.03, 'num_leaves': 128, 'feature_fraction': 0.9, 'min_data_in_leaf': 3, 'two_round': True, 'seed_value': 0, 'AG_args': {'model_type': 'GBM', 'name_suffix': 'Custom', 'disable_in_hpo': True}}]}}\n",
      "Plot summary of models saved to file: agModels-predictReadmit/SummaryOfModels.html\n",
      "*** End of fit() summary ***\n"
     ]
    }
   ],
   "source": [
    "results = predictor.fit_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this summary, we can see that AutoGluon trained many different types of models as well as an ensemble of the best-performing models, which combines their predictions in a weighted manner. Each type of model has various hyperparameters, which the user would traditionally have to specify. \n",
    "Here, AutoGluon is simply using fixed default values for these hyperparameters which are known to typically work well (the **LightGBMClassifierCustom** model here is a second **LightGBMClassifier** where these two models use very different default hyperparameter configurations that are both known to work well). \n",
    "\n",
    "The summary also describes the actual models that were trained during fit and how well each model performed on the held-out validation data (`score_val` column, which lists *accuracy* in this case), how long the model took to produce predictions on the validation data (`pred_time_val`), and how long the model was trained for (`fit_time`). \n",
    "The `stack_level` column distinguishes models which operate on the data directly (`stack_level = 0`) vs. models which combine the predictions of (multiple) other models (`stack_level = 1`) like the weighted-ensemble. \n",
    "\n",
    "We can also perform a similar evaluation of each trained model over our test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ExtraTreesClassifierEntr</td>\n",
       "      <td>0.548333</td>\n",
       "      <td>0.591667</td>\n",
       "      <td>0.199204</td>\n",
       "      <td>0.141177</td>\n",
       "      <td>0.723158</td>\n",
       "      <td>0.199204</td>\n",
       "      <td>0.141177</td>\n",
       "      <td>0.723158</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ExtraTreesClassifierGini</td>\n",
       "      <td>0.546667</td>\n",
       "      <td>0.566667</td>\n",
       "      <td>0.313916</td>\n",
       "      <td>0.132281</td>\n",
       "      <td>0.739485</td>\n",
       "      <td>0.313916</td>\n",
       "      <td>0.132281</td>\n",
       "      <td>0.739485</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NeuralNetClassifier</td>\n",
       "      <td>0.545000</td>\n",
       "      <td>0.508333</td>\n",
       "      <td>0.339290</td>\n",
       "      <td>0.076282</td>\n",
       "      <td>6.777301</td>\n",
       "      <td>0.339290</td>\n",
       "      <td>0.076282</td>\n",
       "      <td>6.777301</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierGini</td>\n",
       "      <td>0.535000</td>\n",
       "      <td>0.591667</td>\n",
       "      <td>0.142991</td>\n",
       "      <td>0.131530</td>\n",
       "      <td>0.956434</td>\n",
       "      <td>0.142991</td>\n",
       "      <td>0.131530</td>\n",
       "      <td>0.956434</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LightGBMClassifier</td>\n",
       "      <td>0.530000</td>\n",
       "      <td>0.650000</td>\n",
       "      <td>0.040716</td>\n",
       "      <td>0.031892</td>\n",
       "      <td>6.098786</td>\n",
       "      <td>0.040716</td>\n",
       "      <td>0.031892</td>\n",
       "      <td>6.098786</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RandomForestClassifierEntr</td>\n",
       "      <td>0.528333</td>\n",
       "      <td>0.608333</td>\n",
       "      <td>0.140743</td>\n",
       "      <td>0.139243</td>\n",
       "      <td>0.808625</td>\n",
       "      <td>0.140743</td>\n",
       "      <td>0.139243</td>\n",
       "      <td>0.808625</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.523333</td>\n",
       "      <td>0.658333</td>\n",
       "      <td>0.360978</td>\n",
       "      <td>0.165535</td>\n",
       "      <td>7.317772</td>\n",
       "      <td>0.006346</td>\n",
       "      <td>0.001362</td>\n",
       "      <td>0.479501</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>CatboostClassifier</td>\n",
       "      <td>0.518333</td>\n",
       "      <td>0.616667</td>\n",
       "      <td>0.052483</td>\n",
       "      <td>0.036838</td>\n",
       "      <td>42.824128</td>\n",
       "      <td>0.052483</td>\n",
       "      <td>0.036838</td>\n",
       "      <td>42.824128</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>LightGBMClassifierCustom</td>\n",
       "      <td>0.493333</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.034328</td>\n",
       "      <td>0.033570</td>\n",
       "      <td>2.430807</td>\n",
       "      <td>0.034328</td>\n",
       "      <td>0.033570</td>\n",
       "      <td>2.430807</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>KNeighborsClassifierDist</td>\n",
       "      <td>0.448333</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.110443</td>\n",
       "      <td>0.105541</td>\n",
       "      <td>0.004107</td>\n",
       "      <td>0.110443</td>\n",
       "      <td>0.105541</td>\n",
       "      <td>0.004107</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>KNeighborsClassifierUnif</td>\n",
       "      <td>0.421667</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.108254</td>\n",
       "      <td>0.107808</td>\n",
       "      <td>0.006475</td>\n",
       "      <td>0.108254</td>\n",
       "      <td>0.107808</td>\n",
       "      <td>0.006475</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         model  score_test  score_val  pred_time_test  \\\n",
       "0     ExtraTreesClassifierEntr    0.548333   0.591667        0.199204   \n",
       "1     ExtraTreesClassifierGini    0.546667   0.566667        0.313916   \n",
       "2          NeuralNetClassifier    0.545000   0.508333        0.339290   \n",
       "3   RandomForestClassifierGini    0.535000   0.591667        0.142991   \n",
       "4           LightGBMClassifier    0.530000   0.650000        0.040716   \n",
       "5   RandomForestClassifierEntr    0.528333   0.608333        0.140743   \n",
       "6      weighted_ensemble_k0_l1    0.523333   0.658333        0.360978   \n",
       "7           CatboostClassifier    0.518333   0.616667        0.052483   \n",
       "8     LightGBMClassifierCustom    0.493333   0.600000        0.034328   \n",
       "9     KNeighborsClassifierDist    0.448333   0.533333        0.110443   \n",
       "10    KNeighborsClassifierUnif    0.421667   0.500000        0.108254   \n",
       "\n",
       "    pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  \\\n",
       "0        0.141177   0.723158                 0.199204                0.141177   \n",
       "1        0.132281   0.739485                 0.313916                0.132281   \n",
       "2        0.076282   6.777301                 0.339290                0.076282   \n",
       "3        0.131530   0.956434                 0.142991                0.131530   \n",
       "4        0.031892   6.098786                 0.040716                0.031892   \n",
       "5        0.139243   0.808625                 0.140743                0.139243   \n",
       "6        0.165535   7.317772                 0.006346                0.001362   \n",
       "7        0.036838  42.824128                 0.052483                0.036838   \n",
       "8        0.033570   2.430807                 0.034328                0.033570   \n",
       "9        0.105541   0.004107                 0.110443                0.105541   \n",
       "10       0.107808   0.006475                 0.108254                0.107808   \n",
       "\n",
       "    fit_time_marginal  stack_level  can_infer  \n",
       "0            0.723158            0       True  \n",
       "1            0.739485            0       True  \n",
       "2            6.777301            0       True  \n",
       "3            0.956434            0       True  \n",
       "4            6.098786            0       True  \n",
       "5            0.808625            0       True  \n",
       "6            0.479501            1       True  \n",
       "7           42.824128            0       True  \n",
       "8            2.430807            0       True  \n",
       "9            0.004107            0       True  \n",
       "10           0.006475            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_perf = predictor.leaderboard(test_data, silent=True)\n",
    "display(model_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the performance of many models decreased on the test data, because even though the validation data was held-out during the optimization of individual models, certain training decisions were still made on the basis of this validation-set thus leading to slight overfitting. This includes decisions like: when to early-stop iteratively trained models like the neural network and gradient boosted trees, as well as what weights to use for combining different models' predictions in the weighted-ensemble. The more decisions that are made based on validation data, the less reliable performance-estimates obtained from this data become (particularly under extensive hyperparameter-tuning). This is why it's critical to only present the test data *after* training has completed to obtain an unbiased estimate of the true predictive performance.\n",
    "\n",
    "For posterity, we show how to predict class-probabilities rather than class-labels with AutoGluon. These are often desired in settings where we want to: gauge the confidence in predictions, rank individuals (here patients based on their estimated readmission probability), or select a different probability-threshold to use in decision-making (eg. if certain types of misclassification errors are more costly than others)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>&lt;30</th>\n",
       "      <th>&gt;30</th>\n",
       "      <th>NO</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.106860</td>\n",
       "      <td>0.069708</td>\n",
       "      <td>0.823432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.053784</td>\n",
       "      <td>0.384020</td>\n",
       "      <td>0.562196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.465354</td>\n",
       "      <td>0.111395</td>\n",
       "      <td>0.423251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.529098</td>\n",
       "      <td>0.210637</td>\n",
       "      <td>0.260265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.011000</td>\n",
       "      <td>0.181537</td>\n",
       "      <td>0.807464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>595</th>\n",
       "      <td>0.030304</td>\n",
       "      <td>0.697547</td>\n",
       "      <td>0.272150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>0.043840</td>\n",
       "      <td>0.168644</td>\n",
       "      <td>0.787517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>597</th>\n",
       "      <td>0.044705</td>\n",
       "      <td>0.682842</td>\n",
       "      <td>0.272454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>598</th>\n",
       "      <td>0.110309</td>\n",
       "      <td>0.358272</td>\n",
       "      <td>0.531419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>599</th>\n",
       "      <td>0.047639</td>\n",
       "      <td>0.246947</td>\n",
       "      <td>0.705414</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>600 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          <30       >30        NO\n",
       "0    0.106860  0.069708  0.823432\n",
       "1    0.053784  0.384020  0.562196\n",
       "2    0.465354  0.111395  0.423251\n",
       "3    0.529098  0.210637  0.260265\n",
       "4    0.011000  0.181537  0.807464\n",
       "..        ...       ...       ...\n",
       "595  0.030304  0.697547  0.272150\n",
       "596  0.043840  0.168644  0.787517\n",
       "597  0.044705  0.682842  0.272454\n",
       "598  0.110309  0.358272  0.531419\n",
       "599  0.047639  0.246947  0.705414\n",
       "\n",
       "[600 rows x 3 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_predprob = predictor.predict_proba(test_data)\n",
    "display(pd.DataFrame(y_predprob,columns=predictor.class_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we call `predict()`, AutoGluon automatically predicts with the model that displayed the best validation performance (i.e. the weighted-ensemble). We can instead specify which model to use for predictions like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['>30', 'NO', 'NO', 'NO', 'NO', '>30', '>30', 'NO', 'NO', 'NO',\n",
       "       '>30', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict(test_data[:20], model='NeuralNetClassifier')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Formats:** \n",
    "AutoGluon can currently operate on data tables already loaded into Python as [pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html), or those stored in files of [CSV format](https://en.wikipedia.org/wiki/Comma-separated_values) or [Parquet format](https://databricks.com/glossary/what-is-parquet). If your data live in multiple tables, you will first need to [join](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html) them into a single table whose rows correspond to statistically independent observations (datapoints) and columns correspond to different features (aka. variables/covariates). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "Erickson et al. [**AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data**](https://arxiv.org/abs/2003.06505). *Arxiv*, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
